Learning with AI Language Models: Guidelines for the Development and Scoring of Medical Questions for Higher Education.

In medical and biomedical education, traditional teaching methods often struggle to engage students and promote critical thinking. The use of AI language models has the potential to transform teaching and learning practices by offering an innovative, active learning approach that promotes intellectual curiosity and deeper understanding. To effectively integrate AI language models into biomedical education, it is essential for educators to understand the benefits and limitations of these tools and how they can be employed to achieve high-level learning outcomes.This article explores the use of AI language models in biomedical education, focusing on their application in both classroom teaching and learning assignments. Using the SOLO taxonomy as a framework, I discuss strategies for designing questions that challenge students to exercise critical thinking and problem-solving skills, even when assisted by AI models. Additionally, I propose a scoring rubric for evaluating student performance when collaborating with AI language models, ensuring a comprehensive assessment of their learning outcomes.AI language models offer a promising opportunity for enhancing student engagement and promoting active learning in the biomedical field. Understanding the potential use of these technologies allows educators to create learning experiences that are fit for their students' needs, encouraging intellectual curiosity and a deeper understanding of complex subjects. The application of these tools will be fundamental to provide more effective and engaging learning experiences for students in the future.

[Open educational resources for otorhinolaryngology : A pilot study on needs assessment and implementation]. / Open Educational Resources für die HNO-Heilkunde : Ein Pilotprojekt zur Bedarfsanalyse und Implementierung.

BACKGROUND: Open educational resources (OER) are educational materials licensed openly by authors, permitting usage, redistribution, and in some instances, modification. OER platforms thereby serve as a medium for distributing and advancing teaching materials and innovative educational methodologies. OBJECTIVE: This study aims to determine the present state of OER in otorhinolaryngology and to examine the prerequisites for seamlessly integrating OER into the curricular teaching of medical schools, specifically through the design of two OER blended learning modules. METHODS: OER content in the field of otorhinolaryngology was analyzed on OER platforms, ensuring its relevance to the German medical curriculum. Data protection concerns were addressed with legal counsel. The blended learning modules were developed in collaboration with medical students and subsequently published as OER. RESULTS AND CONCLUSION: This project yielded the first OER from a German ENT department, tailored to the German medical curriculum. One significant barrier to OER use in medicine, more than in other fields, is data protection. This challenge can be navigated by obtaining consent to publish patient data as OER. OER hold the promise to play a pivotal role in fostering cooperation and collaboration among educators, aiding educators in lesson preparation, and simultaneously enhancing didactic quality.

Using Mixed Reality Simulation to Improve Junior Medical Trainees' Preparedness to Manage High-Acuity Trauma.

High-acuity trauma necessitates experienced and rapid intervention to prevent patient harm. However, upskilling junior trainees through hands-on management of real trauma cases is rarely feasible without compromising patient safety. This quality education report sought to investigate whether a simulation course operated via mixed reality (MR) headset devices (Microsoft HoloLens) could enhance the clinical knowledge recall and preparedness to practice of junior trainees with no prior experience managing trauma.The Plan-Do-Study-Act quality improvement method was used to refine six emergency trauma vignettes compatible with an MR teaching platform. Each vignette was curated by a multidisciplinary team of orthopaedic surgeons, clinical fellows and experts in simulation-based medical education. As a baseline assessment, a 2-hour emergency trauma course was delivered using traditional didactic methods to a cohort of pre-registration medical students with no clinical exposure to high-acuity trauma (n=16). Next, we delivered the MR simulation to an equivalent cohort (n=32). Clinical knowledge scores derived from written test papers were recorded for each group during and 2 weeks after each course. Each attendee's end-of-rotation clinical supervisor appraisal grade was recorded, as determined by a consultant surgeon who supervised participants during a 2-week placement on a major trauma ward. Balancing measures included participant feedback and validated cognitive load questionnaires (National Aeronautics and Space Administration-Task Load Index).Overall, attendees of the MR simulation course achieved and sustained higher clinical knowledge scores and were more likely to receive a positive consultant supervisor appraisal. This project serves as a proof of concept that MR wearable technologies can be used to improve clinical knowledge recall and enhance the preparedness to practice of novice learners with otherwise limited clinical exposure to high-acuity trauma.

The Tree of Knowledge.

In this narrative medicine essay, a medical student uses the analogy of a mature tree to describe the difference in knowledge of his medical school teachers compared with sapling first-year medical students.

Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education.

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

Design and Implementation of a National Program of Assessment Model - Integrating Entrustable Professional Activity Assessments in Canadian Specialist Postgraduate Medical Education.

Traditional approaches to assessment in health professions education systems, which have generally focused on the summative function of assessment through the development and episodic use of individual high-stakes examinations, may no longer be appropriate in an era of competency based medical education. Contemporary assessment programs should not only ensure collection of high-quality performance data to support robust decision-making on learners' achievement and competence development but also facilitate the provision of meaningful feedback to learners to support reflective practice and performance improvement. Programmatic assessment is a specific approach to designing assessment systems through the intentional selection and combination of a variety of assessment methods and activities embedded within an educational framework to simultaneously optimize the decision-making and learning function of assessment. It is a core component of competency based medical education and is aligned with the goals of promoting assessment for learning and coaching learners to achieve predefined levels of competence. In Canada, postgraduate specialist medical education has undergone a transformative change to a competency based model centred around entrustable professional activities (EPAs). In this paper, we describe and reflect on the large scale, national implementation of a program of assessment model designed to guide learning and ensure that robust data is collected to support defensible decisions about EPA achievement and progress through training. Reflecting on the design and implications of this assessment system may help others who want to incorporate a competency based approach in their own country.

Assessing the Landscape in Medical Education Literature in Medical Oncology: A Scoping Review.

PURPOSE: Medical oncology and medical education (ME) have both expanded exponentially over the past 50 years; thus, it is important to understand the current status of postgraduate medical oncology education and develop ways to advance this field. This study undertakes a scoping review of ME literature in medical oncology to inform future scholarship in this area. METHODS: MEDLINE, Embase, ERIC, and Web of Science were searched to find peer-reviewed English language articles on postgraduate ME in medical oncology published from 2009 to 2020. Established scoping review methodologies were used in study design; articles were classified by specialty, learner training level, region of authorship, institution type, year of publication, journal type, study methodology, and research topic. Curriculum intervention, scholarship, and domain(s) of physician competency were also assessed. The results were interpreted using descriptive statistics and collated using predetermined conceptual frameworks. RESULTS: A total of 2,959 references were initially found across four databases. After title and abstract screening, 305 articles remained; after full-text review, 144 articles were included in final analysis. Postgraduate medical oncology education research is increasing, with the majority of articles published in North America. Quantitative studies were most common, primarily survey approaches. For physician competencies, professionalism and medical expertise comprised the large majority of article focuses, whereas very few articles addressed leadership or health advocacy. Curriculum development, professional development, and communication skills were dominant research themes while no articles discussed teacher training. CONCLUSION: Although areas such as professionalism and communication skills are well-studied, medical oncology ME research is lacking in leadership, health advocacy, and teacher training. This study provides valuable guidance for future ME research in medical oncology and establishes a benchmark to examine changes in educational scholarship over time.

Effectiveness of online training in improving primary care doctors' competency in brief tobacco interventions: A cluster-randomized controlled trial of WHO modules in Delta State, Nigeria.

BACKGROUND: The World Health Organization (WHO) strongly recommends that brief tobacco interventions should be routinely offered in primary care. However, medical doctors do not consistently and effectively intervene during their encounters with cigarette smokers. There is a paucity of studies assessing the effect of training on the tobacco intervention competency of primary care doctors in Nigeria. AIM: To evaluate the effectiveness of online training in improving competency in brief tobacco interventions among primary care doctors in Delta State, Nigeria. METHODS: A cluster-randomized controlled trial was conducted among eligible doctors working in government-owned facilities. The 22 eligible Local Government Areas (LGAs) served as clusters. The intervention group received a WHO six-hour online course on brief tobacco cessation intervention, delivered via Zoom. The control group received no intervention. A structured questionnaire was sent to participants via WhatsApp before and six months after the training. The primary outcome variables were scores for knowledge, attitude, self-efficacy, and practice. Differences in change of scores between intervention and control groups were assessed with t-test. To adjust for clustering, these inter-group differences were further analyzed using linear mixed-effects regression modeling with study condition modeled as a fixed effect, and LGA of practice entered as a random effect. RESULTS: The intervention group had a significantly higher mean of change in scores for knowledge (effect size 0.344) and confidence (effect size 0.52). CONCLUSION: The study shows that training, even online, positively affects clinician competency in brief tobacco intervention. This is important for primary care systems in developing countries. Mandatory in-service training and promotion of the WHO modules are recommended.

Comparing the effects of team-based and problem-based learning strategies in medical education: a systematic review.

BACKGROUND: Recently, there has been a concerted effort within medical schools to depart from conventional lecture-based learning approaches to alternative teaching methods such as team-based learning (TBL) and problem-based learning (PBL), with the aim of enhancing both student engagement and instructional efficacy. Despite this shift, a comprehensive review that directly compares the impacts of PBL and TBL methods in medical education is lacking. This study seeks to address this gap by conducting a meta-analysis that compares the effects of TBL and PBL in the context of medical education. METHODS: Studies from Embase, PubMed, Web of Science, China National Knowledge Infrastructure, and Chinese Wanfang Database were searched, from inception to July 11, 2023. A meta-analysis was performed using Stata 14.0, and a total of 10 studies (including 752 participants) were included. The standardized mean difference (SMD) was used to estimate pooled effects. Heterogeneity was detected using the I2 statistic and further explored using meta-regression analysis. RESULTS: Compared with PBL, TBL significantly increased the number of theoretical tests (SMD = 0.37, 95% CI: 0.02-0.73). Additionally, TBL greatly improved teamwork skills compared with PBL. However, there were no significant differences between the TBL and PBL groups concerning practical skill scores, learning interest, or understanding skills. CONCLUSION: TBL in the theoretical aspects of medical education appears to be more effective than PBL in improving theoretical test scores and teamwork skills, providing evidence for the implementation of TBL in medical education.

Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions.

PROBLEM: Creating medical exam questions is time consuming, but well-written questions can be used for test-enhanced learning, which has been shown to have a positive effect on student learning. The automated generation of high-quality questions using large language models (LLMs), such as ChatGPT, would therefore be desirable. However, there are no current studies that compare students' performance on LLM-generated questions to questions developed by humans. APPROACH: The authors compared student performance on questions generated by ChatGPT (LLM questions) with questions created by medical educators (human questions). Two sets of 25 multiple-choice questions (MCQs) were created, each with 5 answer options, 1 of which was correct. The first set of questions was written by an experienced medical educator, and the second set was created by ChatGPT 3.5 after the authors identified learning objectives and extracted some specifications from the human questions. Students answered all questions in random order in a formative paper-and-pencil test that was offered leading up to the final summative neurophysiology exam (summer 2023). For each question, students also indicated whether they thought it had been written by a human or ChatGPT. OUTCOMES: The final data set consisted of 161 participants and 46 MCQs (25 human and 21 LLM questions). There was no statistically significant difference in item difficulty between the 2 question sets, but discriminatory power was statistically significantly higher in human than LLM questions (mean = .36, standard deviation [SD] = .09 vs mean = .24, SD = .14; P = .001). On average, students identified 57% of question sources (human or LLM) correctly. NEXT STEPS: Future research should replicate the study procedure in other contexts (e.g., other medical subjects, semesters, countries, and languages). In addition, the question of whether LLMs are suitable for generating different question types, such as key feature questions, should be investigated.

The Promise and Perils of Artificial Intelligence in Health Professions Education Practice and Scholarship.

ABSTRACT: Artificial intelligence (AI) methods, especially machine learning and natural language processing, are increasingly affecting health professions education (HPE), including the medical school application and selection processes, assessment, and scholarship production. The rise of large language models over the past 18 months, such as ChatGPT, has raised questions about how best to incorporate these methods into HPE. The lack of training in AI among most HPE faculty and scholars poses an important challenge in facilitating such discussions. In this commentary, the authors provide a primer on the AI methods most often used in the practice and scholarship of HPE, discuss the most pressing challenges and opportunities these tools afford, and underscore that these methods should be understood as part of the larger set of statistical tools available.Despite their ability to process huge amounts of data and their high performance completing some tasks, AI methods are only as good as the data on which they are trained. Of particular importance is that these models can perpetuate the biases that are present in those training datasets, and they can be applied in a biased manner by human users. A minimum set of expectations for the application of AI methods in HPE practice and scholarship is discussed in this commentary, including the interpretability of the models developed and the transparency needed into the use and characteristics of such methods.The rise of AI methods is affecting multiple aspects of HPE including raising questions about how best to incorporate these models into HPE practice and scholarship. In this commentary, we provide a primer on the AI methods most often used in HPE and discuss the most pressing challenges and opportunities these tools afford.

Untapped resource: the simulation-based healthcare environment as a means to study human stress.

The effects of 'stress' within the healthcare professions are wide-reaching, not least of all within the field of simulation-based healthcare education. Whilst this popular method of experiential learning offers a 'safe space' for participants to develop their skillset, it also has a more surreptitious action; namely, the incubation of simulation-related stress. Currently, research concerning the complex relationship between stress, learning, and performance is ambiguous, leaving fertile ground for simulationists to debate what level of stress is appropriate for an optimised educational experience. In this narrative review, we examine the human response to stress and outline the various methods that have been used by researchers to measure stress in a quantifiable and standardised way. We then provide a brief overview of simulation-based healthcare education before describing why stress responses have been of interest to healthcare educationalists for some time. Finally, we outline how simulation education environments might provide an ideal environment for studying the human response to stress generally, with ramifications extending beyond the field of medical education.

Mapping the Terrain of Faculty Development for Simulation: A Scoping Review.

ABSTRACT: Understanding what interventions and approaches are currently being used to improve the knowledge, skills, and effectiveness of instructors in simulation-based education is an integral step for carving out the future of simulation. The current study is a scoping review on the topic, to uncover what is known about faculty development for simulation-based education.We screened 3259 abstracts and included 35 studies in this scoping review. Our findings reveal a clear image that the landscape of faculty development in simulation is widely diverse, revealing an array of foundations, terrains, and peaks even within the same zone of focus. As the field of faculty development in simulation continues to mature, we would hope that greater continuity and cohesiveness across the literature would continue to grow as well. Recommendations provided here may help provide the pathway toward that aim.

Thawed Porcine Model for Surgical Training: A Novel Conservative Procedure.

Background: General Surgery course is a mandatory in medical schools and continuing surgery training is important even to experienced surgeons which they need to maintain and/or improve their surgical skills. Additionally, the models used for that practice are human cadavers, anesthetized porcine, or simulators and are not accessible for medicine schools or physicians in many countries. Therefore, we present a new technical procedure for preparation of frozen experimental animal's cadavers for medical surgical training. Materials and Methods: To perform the study, one porcine slaughtered and frozen at -20°C was used. The porcine cadaver was thawed at room temperature (25°C) and then the pneumoperitoneum test was performed and viscera inspection carried out. Results: The porcine cadaver took 20 hours to completely thaw. The pneumoperitoneum was successfully performed with total distention of the abdominal cavity. All viscera were well preserved maintaining important in vivo characteristics for consistency. Conclusion: The use of thawed porcine cadaver as a model to train many surgical procedures including videolaparoscopy is feasible. The tissues were well preserved by this method and was financially accessible and could be used for different techniques, equipment, and material tests.

Powerful medical education improves health care quality and return on investment.

INTRODUCTION: Powerful medical education (PME) involves the use of new technologies informed by the science of expertise that are embedded in laboratories and organizations that value evidence-based education and support innovation. This contrasts with traditional medical education that relies on a dated apprenticeship model that yields uneven results. PME involves an amalgam of features, conditions and assumptions, and contextual variables that comprise an approach to developing clinical competence grounded in education impact metrics including efficiency and cost-effectiveness. METHODS: This article is a narrative review based on SANRA criteria and informed by realist review principles. The review addresses the PME model with an emphasis on mastery learning and deliberate practice principles drawn from the new science of expertise. Pub Med, Scopus, and Web of Science search terms include medical education, the science of expertise, mastery learning, translational outcomes, cost effectiveness, and return on investment. Literature coverage is comprehensive with selective citations. RESULTS: PME is described as an integrated set of twelve features embedded in a group of seven conditions and assumptions and four context variables. PME is illustrated via case examples that demonstrate improved ventilator patient management learning outcomes compared to traditional clinical education and mastery learning of breaking bad news communication skills. Evidence also shows that PME of physicians and other health care providers can have translational, downstream effects on patient care practices, patient outcomes, and return on investment. Several translational health care quality improvements that derive from PME include reduced infections; better communication among physicians, patients, and families; exceptional birth outcomes; more effective patient education; and return on investment. CONCLUSIONS: The article concludes with challenges to hospitals, health systems, and medical education organizations that are responsible for producing physicians who are expected to deliver safe, effective, and cost-conscious health care.

Deidentifying Narrative Assessments to Facilitate Data Sharing in Medical Education.

PROBLEM: Narrative assessments are commonly incorporated into competency-based medical education programs. However, efforts to share competency-based medical education assessment data among programs to support the evaluation and improvement of assessment systems have been limited in part because of security concerns. Deidentifying assessment data mitigates these concerns, but deidentifying narrative assessments is time-consuming, resource intensive, and error prone. The authors developed and tested a tool to automate the deidentification of narrative assessments and facilitate their review. APPROACH: The authors met throughout 2021 and 2022 to iteratively design, test, and refine the deidentification algorithm and data review interface. Preliminary testing of the prototype deidentification algorithm was performed using narrative assessments from the University of Saskatchewan emergency medicine program. The algorithm's accuracy was assessed by the authors using the review interface designed for this purpose. Formal testing included 2 rounds of deidentification and review by members of the authorship team. Both the algorithm and data review interface were refined during the testing process. OUTCOMES: Authors from 3 institutions, including 3 emergency medicine programs, an anesthesia program, and a surgical program, participated in formal testing. In the final round of review, 99.4% of the narrative assessments were fully deidentified (names, nicknames, and pronouns removed). The results were comparable for each institution and specialty. The data review interface was improved with feedback obtained after each round of review and found to be intuitive. NEXT STEPS: This innovation has demonstrated viability evidence of an algorithmic approach to the deidentification of assessment narratives while reinforcing that a small number of errors are likely to persist. Future steps include the refinement of both the algorithm to improve its accuracy and the data review interface to support additional data set formats.

The change of the COVID-19 pandemic on medical education in Korea: a national survey of medical schools.

PURPOSE: This study aims to investigate how medical schools in Korea managed their academic affairs and student support in response to the coronavirus disease 2019 (COVID-19) pandemic and their plans for the post-COVID-19 era. METHODS: An online survey was conducted, and a link to a questionnaire was distributed to all the administrators of the 40 medical schools in Korea. The final analysis data involved responses from 33 medical schools and 1,342 students. Frequency analysis, cross-tabulation analysis, Fisher's exact test, and one-way analysis of variance were applied for statistical analysis. RESULTS: Regarding instruction methods, most medical schools in Korea transitioned from in-person learning to video-on-demand learning (51.5%) and real-time online learning (42.4%). Among the school leaders, 36.4% planned to continue offering online classes combined with in-person classes beyond the end of the pandemic. Among the students, the online class concentration and participation score was 3.0 points or lower, but the class understanding score was 3.6 points, above a moderate level. CONCLUSION: Students cited the shorter times needed to attend school and being able to take classes repeatedly as advantages of online classes, and over one-third of medical schools intended to continue with a hybrid of in-person and online learning even after the COVID-19 pandemic restrictions end.